Field personnel, such as soldiers, police SWAT teams, and first responders face challenging, dangerous environments, often with little advance knowledge or information about their surroundings. Currently, this Intelligence, Surveillance & Reconnaissance (ISR) information is provided by archival pictures, satellite imagery and prior or second-hand experiences. Although satellite imagery is currently the preferred method for gaining Situational Awareness (SA) in military outdoor environments, it has many shortcomings. As is known by those skilled in the art, situational awareness is defined as the combination of perception of elements in the environment, the comprehension of their meaning, and the projection of their status in the future. Operator situational awareness captures the ability of operators to make effective choices as a function of information processing efficiency. As such, situational awareness can significantly influence human behavior and hence, human-unmanned vehicle system performance. Since situational awareness is dynamic, it can influence operator responses over time and as a result, can dynamically impact supervisory control performance.
Unclassified satellite imagery maps available to field personnel are two dimensional images, with no elevation information and/or fixed points of view. In addition, field personnel may use simple maps that are not provided by satellite imagery. These maps are often outdated, and, due to shadows and shading, give false impressions of elevations and details of the environment. Critical features of buildings, such as doorways and windows are hidden from view. Combined, these flaws often provide field personnel with a false mental model of their environment. As is known by those skilled in the art, a mental model is defined as the cognitive constructed, which is created by a user to aid them in comprehending how a system operates and guiding them in their decision making processes regarding the system.
Given the need of these personnel to simultaneously perform a primary task, such as finding a Person of Interest (POI), as well as exploring the environment, an autonomous robot (i.e., an unmanned vehicle) would allow these groups to better perform ISR and improve their SA in real-time. Recent efforts have led to the creation of Micro Aerial Vehicles (MAVs), a class of Unmanned Aerial Vehicle (UAV), which are small and have autonomous capabilities. An MAV can hover in place, perform Vertical Take-Off and Landing, and easily rotate with a small sensor payload. The compact size of these vehicles and their maneuvering capabilities make them well-suited for performing highly localized ISR missions with an MAV operator working within the same environment as the vehicle. These localized missions require users to devote significant amounts of cognitive resources to give the MAVs gross directions (i.e., going to a building of interest), and then 100% of attention to fine tune control of such a vehicle (i.e., position a vehicle at a window to peer in.)
Unfortunately, existing interfaces for MAVs, and unmanned vehicles in general, ignore the competing needs of field operators, requiring bulky equipment and the full attention of an operator. A majority of interfaces and Ground Control Stations (GCSs) require the full attention of the operator. These systems require extensive training before an operator can safely and effectively operate the MAV. GCSs that allow the operator to manually position and orient the MAV rely on a classical 1st order feedback control loop, which allows operators to directly control the thrust, pitch, and roll/yaw of the MAV. This complex feedback loop demands full attention of the operator, and thereby decreases overall SA of the operator of the environment and task at hand.
Human control of systems that incorporate one or more feedback loops is defined as an Nth order system, where N refers to the derivative of the feedback loop used in the controls. A 1st order feedback loop responds to changes in the first derivative of the system, such as velocity derived from position. Error, the difference between the output and the desired state, is fed back to the input in an attempt to bring the output closer to the desired state. For example, changing the heading of an MAV via a first-order feedback loop requires constantly changing the rate of yaw and roll of the MAV until the desired heading is reached. Typically, this is executed by humans as a pulse input that requires at least two distinct actions, namely, starting the turn and then ending the turn. In contrast, with a 0th order control loop, an operator simply provides a command with the desired heading, such as, South, and the vehicle autonomously turns to this heading. A 1st order system requires more attention by the operator as compared to a 0th order system since he or she must continually oversee the vehicle in order to stop, such as, but not limited to, a turn, at the right time.
A 2nd order control loop relies on changing the acceleration of the system. It is noted that inherently, UAV systems are 2nd order systems. It is generally recognized that humans have significant difficulty controlling 2nd order and higher systems. Due to the increased complexity of the feedback loops and number of actions required to successfully complete a maneuver, the cognitive workload of an operator is significantly higher for 2nd order systems than when operating 0th or 1st order controls, leading to lower performance. Teleoperation, defined as the remote control of an unmanned vehicle via some set of external controls and displays, only exacerbates these problems because additional communication latencies are introduced into the system, in addition to the lack of sensory perception on the part of the operator, who is not physically present. When a human actor acts based on a GCS view that is delayed due to system latency, the result is often accidents involving the UAV.
While human operators are thought to be effective 1st order controllers, it is doubtful whether UAV operators can effectively execute 2nd order control of UAVs. System communication delays, the lack of critical perceptual cues, and the need for extensive training, which can result in pilot-induced oscillations and inappropriate control responses, suggest that 1st order control is a poor approach to any type of UAV (and unmanned vehicle) control. This problem is likely more serious for MAV operators who are not, by the nature of their field presence, able to devote the necessary cognitive resources needed to fully attend to the control dynamics of the MAV.
By comparison, a 0th order control loop significantly reduces the workload because the operator does not need to continually monitor the movement of the vehicle, such as, but not limited to, as the vehicle turns to a new heading, however, there is some cost in vehicle maneuverability. For operating a MAV, 0th order interfaces represent the highest degree of safety because users are not prone to errors as operators try to calculate the position of the vehicle.
Therefore, it is desirable to have an interface that can allow an operator to easily control an unmanned vehicle at a high-level supervisory mode of interaction for general commands, as well as a fine-grained, lower level of control when more nuanced actions are required. Such control should not require complete attention of operators, nor require extensive training.